Bayesian Model Comparison and Validation

Models are the venue for much of the work of the economics profession. We use them to express, compare and evaluate alternative ways of addressing important questions. Applied econometricians are called upon to engage in these exercises using data and, often, formal methods whose properties are understood in decision-making contexts. This is true of work in other sciences as well. There is an enormous literature on alternative formal approaches to these tasks, and in particular on the relative advantages of Bayesian and frequentist methods. By “Bayesian” I mean statistical inference that reaches a conclusion by means of a conditional distribution of unknown quantities given known quantities and model specifications. This conditional distribution follows from applying Bayes’s theorem to the joint distribution of known and unknown quantities in the model specifications. Known quantities, including data, are treated as observed values of random variables. Unknown quantities, including functions of parameters and as yet unobserved data, are treated as unobserved random variables. By “frequentist” I mean inference derived from the distribution of statistics in repeated sampling. This includes the use of hypothesis tests, confidence intervals and p-values. The literature on the relative advantages of Bayesian and frequentist methods is also very large. Bayesian procedures coincide with models of rational behavior, especially updating information and behavior in uncertain environments, and Bayesian learning is the dominant paradigm in formal modeling of this behavior. It is easy to concoct interesting situations in which Bayesian methods yield reasonable outcomes and frequentist methods do not. On the other hand, Bayesian learning assumes costless cognition, whereas in fact reasonable specification of the joint distribution of known and unknown quantities can be very demanding. Frequentist methods avoid many of these demands, and for this reason they are likely to enjoy continued widespread use even as advances in simulation methods continue to widen the scope of application for Bayesian methods in econometrics and statistics. In this context the relative advantages of Bayesian and frequentist approaches seem clear. This has been reaffirmed in a rich literature that includes Box (1980), Dawid (1982), Rubin (1984), Gelman et al. (1996) and Little (2006). An informal statement of these conclusions helps to explain the organization of the rest of this essay, which will then provide a more precise rendering. Suppose, first, that conclusions are to be drawn based on a workably small number of well-articulated models. Then the situation for Bayesian inference is not fundamentally altered whether that number is one or several, so long as prior distributions are extended to assign prior probabilities to the respective models. One must (and can) deal with some interesting technical details, but the paradigms of rational behavior and Bayesian learning emerged unscathed. Indeed, they differ little operationally, in